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Abstract 


This paper presents an integrated image fusion and match score fusion of multispectral face images. The fusion of visible and long wave 
infrared face images is performed using 2v-granular SVM which uses multiple SVMs to learn both the local and global properties of the 
multispectral face images at different granularity levels and resolution. The 2v-GSVM performs accurate classification which is subsequently 
used to dynamically compute the weights of visible and infrared images for generating a fused face image. 2D log polar Gabor transform and 
local binary pattern feature extraction algorithms are applied to the fused face image to extract global and local facial features, respectively. 
The corresponding match scores are fused using Dezert Smarandache theory of fusion which is based on plausible and paradoxical reasoning. 
The efficacy of the proposed algorithm is validated using the Notre Dame and Equinox databases and is compared with existing statistical, 


learning, and evidence theory based fusion algorithms. 
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1. Introduction 


Current face recognition systems capture faces of cooper- 
ative individuals in a controlled environment as part of the 
face recognition process. It is therefore possible to control the 
lighting, pose, background, and quality of images. Under these 
conditions, the performance of face recognition algorithms is 
greatly enhanced. However, there is still a need for more robust 
and efficient face recognition algorithms to address challenges 
such as changes in illumination, variations in pose and expres- 
sion, variations in facial features due to aging, and altered ap- 
pearances due to disguise [1]. 

Face recognition algorithms generally use visible spectrum 
images for recognition because they provide clear representa- 
tion of facial features and face texture to differentiate between 
two individuals. However, visible spectrum images also pos- 
sess several other properties which affect the performance of 
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recognition algorithms. For example, changes in lighting affect 
the representation of visible spectrum images and can influ- 
ence feature extraction. Other variations in face images such as 
facial hairs, wrinkles, and expression are also evident in vis- 
ible spectrum images and these variations increase the false 
rejection rate of face recognition algorithms. To address the 
challenges posed by visible spectrum images, researchers have 
used infrared images for face recognition [2-5]. Among all the 
infrared spectrum images, long wave infrared (LWIR) images 
possess several properties that are complementary to visible 
images. LWIR or thermal images are captured in the range of 
8—12 um. These images represent the heat pattern of the object 
and are invariant to illumination and expression. Face images 
captured in LWIR spectrum have less intra-class variation and 
help to reduce the false rejection rate of recognition algorithms. 
These properties of LWIR and visible images can be combined 
to improve the performance of face recognition algorithms. 

In literature, researchers have compared the performance 
of visible and thermal face recognition using several face 
recognition algorithms. These results show that for variation 
in expression and illumination, thermal images provide better 
recognition performance compared to visible images [2,6,7]. 
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Further, several fusion algorithms have been proposed to fuse 
the information extracted from visible and LWIR face im- 
ages at image level [8—11], feature level [10-12], match score 
level [12], and decision level [12]. Information fusion of mul- 
tispectral images provides better performance compared to ei- 
ther visible or infrared spectrum images. However, research in 
multispectral information fusion is relatively new and intel- 
ligent techniques such as granular computing, support vec- 
tor machine (SVM), and theory of evidence are not explored. 
These intelligent techniques can enhance the recognition perfor- 
mance by providing better generalization capabilities to handle 
imprecise information. 

In this paper, we propose algorithms to fuse LWIR and vis- 
ible face images at image level and match score level. We first 
propose the formulation of 2v-granular SVM (2v-GSVM) for 
pattern classification which is used in the proposed image fu- 
sion algorithm. The proposed image fusion algorithm learns the 
properties of the multispectral face images at different resolu- 
tion and granularity levels to determine optimal information and 
combines them to generate a fused image. We then apply 2D log 
polar Gabor [13] and local binary pattern [14] face recognition 
algorithms to extract global and local features from the fused 
image. The match scores obtained by matching these features 
are fused using the proposed Dezert Smarandache (DSm) fu- 
sion algorithm [15,16]. The proposed DSm match score fusion 
algorithm is based on evidence theory which performs fusion 
depending on the evidence and belief of the match scores. On 
the Notre Dame [17,18] and Equinox [19] face databases, this 
integrated hierarchical scheme yields verification accuracy of 
more than 99.5%. 

We have organized the proposed algorithms in four sections. 
Section 2 describes the proposed formulation of 2v-GSVM. 
In Section 3, we describe the proposed visible and infrared 
face image fusion algorithm which dynamically and locally 
computes the weights using 2v-GSVM to generate the fused 
face image. We then describe the overview of DSm theory and 
the proposed DSm match score fusion algorithm in Section 4. 
Finally in Section 5, we combine the two levels of fusion to 
present the integrated multilevel image fusion and match score 
fusion algorithm. Section 6 describes the databases and existing 
algorithms used for validation of the proposed algorithms. The 
experimental results are summarized in Section 7. 


2. 2v-granular support vector machine 


SVM is widely used in classification problems because it is 
designed to circumvent the overfitting problem [20-22] and is 
generalized to optimally perform classification on new train- 
ing data. In literature, different variants of SVM have been 
proposed such as SVM, v-SVM, and dual v-SVM (2v-SVM). 
These variants are designed to improve the classification ac- 
curacy and address other challenges such as reduction in time 
complexity and classification with disparate number of train- 
ing samples per class. In our previous research [12], we used 
2v-SVM for feature fusion, match score fusion, and expert fu- 
sion. We observed that 2v-SVM provides better classification 
accuracy compared to classical SVM and is computationally 


more efficient. Recently, Tang et al. [23—26] applied the concept 
of granular computing [27—30] to SVM and proposed GSVM 
which is more adaptive to the data distribution in comparison 
to SVM. Tang et al. have also shown that for several classifi- 
cation applications, GSVM outperforms SVM both in terms of 
classification accuracy and computational time. In this paper, 
we extend the formulation to 2v-GSVM which embodies the 
properties of both GSVM and 2v-SVM. We first describe the 
formulation of 2v-SVM [22] followed by the granular model- 
ing of 2v-SVM. 

Let {x;, yj} be a set of N data vectors with x; € RI, yi € 
(+1, —1),andi=1,..., N. Xx; is the ith data vector that belongs 
to the binary class y;. According to Chew et al. [22] the objective 
of training 2v-SVM is to find the hyperplane that separates two 
classes with the widest margins, i.e., 


we(x) +b=0, (1) 

subject to 

yilwe(xi)+b)2(p— Wi), p, Wi 20 (2) 

to minimize, 

1 

ziw? — Civ — Wi), (3) 
i 


where p is the position of the margin and v is the error param- 
eter. (x) is the mapping function used to map the data space 
to the feature space and provide generalization for the decision 
function that may not be a linear function of the training data. 
Ci(vp — Wi) is the cost of errors, w is the normal vector, b is 
the bias, and y; is the slack variable for classification errors. 
Slack variables are introduced to handle classes which cannot 
be separated by a hyperplane. Let v} and v_ be the error pa- 
rameters for training the positive and negative classes, respec- 
tively. Using these, the error parameter, v, is calculated as 
2V4V_ 


v= ——,  0<v,<1 and 0<v_<l. (4) 
Va VR 


Error penalty C; is defined as 


C4 if yi = +1, 


aei if yi =—1, ©) 


where 


agal 

G= E (: + *)| (6) 
y —1 

C= [»- (: + =)| (7) 
V+ 


and n+ and n— are the number of training points for the posi- 
tive and negative classes, respectively. Further, the Wolfe dual 
formulation of 2v-SVM can be written as 


1 
L=} u- 5 2 ajyiyjK (Hi, xj) ; (8) 
i ij 


where i, j € 1,..., N, «i, «j are the Lagrange multipliers and 
K(-) is the kernel function. Finally, iterative decomposition 
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training based optimization algorithm [22] is used to train the 
2v-GSVM. 

In this paper, we extend the formulation of 2v-SVM by 
using the granular computing approach similar to Ref. [25]. 
Granular computing is a knowledge-oriented divide and con- 
quer approach to problem solving Refs. [27—30]. In granu- 
lar computing, information is divided into subproblems called 
granules and these subproblems are solved individually at dif- 
ferent granularity levels. Using this concept, 2v-GSVM is for- 
mulated as follows. 

Let the complete feature space be divided into k subspaces 
with one 2v-SVM operating on each subspace. The ith 2v-SVM 
is represented by 2vSVM,, where i = 1, 2,...,k. From each 
of the subspace, we obtain the corresponding L; using Eq. (8). 
We then compute the compound margin width W by using all 
the L; values: 


k 
t; 
W= Xo | @vSVM; > Li) = Lal. (9) 


i=1 


where t; is the number of training data in the ith subspace and 
t= aa ti. 2vSVM; :—> Li represents the SVM operating 
on the ith subspace. 2v-SVM learning yields L; at local level, 
and Lo is obtained by learning another 2v-SVM on the com- 
plete feature space at global level. This equation provides the 
margin width associated to a hyperplane. There are different 
methods to divide the feature space and hence different hyper- 
planes associated with each of the granule generation method 
can be obtained. We compute the classification accuracy of all 
the hyperplanes on the training data and then select the hyper- 
plane that optimally classifies the training data. In contrast to a 
single SVM that deals with large parameter space and results 
in large training time, 2v-GSVM uses multiple SVMs to learn 
both the local and global properties of the training data at dif- 
ferent granularity levels. This 2v-GSVM is then used for fusing 
multispectral face images. 


3. Multispectral face image fusion using 2v-GSVM 


In multispectral face recognition, visible images provide the 
reflectance property and LWIR images provide the thermal 
property. In 2v-GSVM based image fusion, we combine these 
properties to generate a fused image which possess both the 
properties and can be used to improve the recognition per- 
formance. Although there are several multispectral face im- 
age fusion algorithms in literature, they have some limitations 
which affect the face recognition performance. Genetic algo- 
rithm based fusion proposed by Bebis et al. [31] suffers from 
making a good choice of fitness function. Fusion algorithm 
proposed by Kong et al. [4] suffers from the empirical con- 
stant weights which are assigned to the wavelet coefficients of 
visible and LWIR images. In real world applications, weights 
should be dynamically and locally assigned for optimal mul- 
tispectral information fusion. In this section, we propose the 
multispectral face image fusion algorithm which dynamically 
and locally computes the weights for fusion using 2v-GSVM. 
Fig. 1 illustrates the steps involved in the proposed image 


fusion algorithm. The algorithm is divided into two steps: im- 
age registration and image fusion. 


3.1. Mutual information based multispectral face image fusion 


Visible and infrared images captured at different time in- 
stances can have variations due to camera angle, expression, 
and geometric deformations. To optimally fuse two multispec- 
tral images, we first need to minimize the linear and non-linear 
differences between the two images. In this section, we pro- 
pose the use of mutual information based registration algorithm 
for registering visible and thermal face images. Mutual infor- 
mation is a concept from information theory in which statis- 
tical dependence is measured between two random variables. 
Researchers in medical imaging have used mutual information 
based registration algorithms to effectively fuse images from 
different modalities such as CT and MRI [32,33]. Registration 
of multispectral face images is described as follows. 

Let V and / be the input visible and infrared face images for 
registration. Mutual information between the two face images 
can be represented as 


M(V,I)=H(V)+ AU) —- AV, 1), (10) 


H(-) is the entropy of the image and H(V, I) is the joint en- 
tropy. Registering V with respect to J requires maximization 
of mutual information between J and V, thus maximizing the 
entropy H(V) and H(I), and minimizing the joint entropy 
H(V, I). Mutual information based registration algorithms are 
sensitive to changes that occur in the distributions as a result of 
difference in overlap regions. To address this issue, Studholme 
et al. [34] proposed normalized mutual information which can 
be represented as 

NM(V, A a A (11) 

H(V,I) 

The registration is performed on a transformation space, T, such 
that 


a b O 
T= |: d o|, (12) 
e f 1 


where a, b, c,d are the shear, scale, and rotation parameters, 
and e, f are the translation parameters. Using the normalized 
mutual information, we define a search strategy to find the 
transformation parameters, T*, by exploring the search space, 
T: 


T* = arg TARIN MU, T(V))}. (13) 


Multispectral face images V and Z are thus registered using 
the transformation parameters T*. This registration algorithm 
is linear in nature. To accommodate the non-linear variation in 
face images, we apply multiresolution image pyramid scheme 
in which we first build Gaussian pyramid of both visible and 
thermal face images. Registration parameters are estimated at 
the coarsest level and used to warp the face image in the next 
level of the pyramid. The process is iteratively repeated through 
each level of the pyramid and a final transformed visible face 
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Fig. 1. Schematic diagram of the proposed 2v-GSVM image fusion algorithm. 
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Fig. 2. Example of visible and infrared image registration on the Notre Dame 
face database [18]. Visible image is registered with respect to the LWIR 
image. Size of detected visible face image is 855 x 1024 and infrared face 
image is 115 x 156. 


image is obtained at the finest pyramid level. In this manner, 
we handle the large global variations at the coarsest resolu- 
tion level and local non-linear variations at the finest resolution 
level. Fig. 2 shows examples of the registration algorithm in 
which visible face image is registered with respect to LWIR 
face image. 


3.2. Proposed 2v-GSVM image fusion algorithm 


In the proposed face image fusion algorithm, registered 
visible and infrared face images are fused using 2v-GSVM 
and discrete wavelet transform (DWT) [35]. The fusion al- 
gorithm uses the activity level, a, of face images which is 
defined as 


where V is the visible face image, and X and Y are the rows 
and columns of the face image, respectively. The proposed fu- 
sion algorithm is divided into two parts: (1) training and (2) 
classification and fusion. 

Training 2v-GSVM: We learn the 2v-GSVM for image fusion 
by using the activity levels of labeled visible and infrared train- 
ing face images. The training algorithm is described as follows: 

Step 1: Visible and infrared training face images are decom- 
posed using DWT to obtain three-level approximation, hori- 
zontal, vertical, and diagonal subbands. 

Step 2: Let Vit}, VLH;» VHL; and Vpy, be the subbands 
of visible face image where j = 1, 2, 3 represents the decompo- 
sition levels. Similarly, let lrt; ILH; IHL; and IHH; be the 
subbands of infrared face image corresponding to each decom- 
position level, j. Each subband of both visible and infrared face 
images is divided into windows of size 8 x 8 and the activity 
level of each window is computed using Eq. (14). 

Step 3: The activity levels of all labeled training face images 
are used as input to 2v-GSVM. In training, two 2v-GSVMs are 
learned, one for visible face images and another for infrared 
face images. 

Step 4: 2v-GSVM trained for visible images classifies the 
activity levels of visible spectrum face images as Good or +1 
and Bad or —1. Similarly, 2v-GSVM trained for infrared face 
images classifies the activity levels of infrared face images into 
Good or Bad class. 

Classification and fusion: We classify the properties of vis- 
ible and infrared face images using trained 2v-GSVMs. This 
classification is used to dynamically compute the weights of 
visible and infrared face images in multispectral image fusion. 

Step 1: Visible and infrared face images of an individual 
are provided as input. Similar to Steps 1 and 2 of the training 
algorithm, both the input face images are decomposed into 








1 X-1Y-1 Y-1X-1 
a= |z| È VED- -DP +) DAVE AD -VE-L AP |, (14) 
i=0 j=1 j=0 i=1 
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three-level DWT and activity levels of 8 x 8 windows are com- 
puted. Let ay and a; be the activity levels computed from vis- 
ible and infrared face images, respectively. 

Step 2: 2v-GSVM classifier is used to classify the activity 
levels of different subbands of visible face images as Good or 
Bad. A binary decision matrix, dy, is generated which contains 
value 1 if the activity level is Good and 0 if the activity level 
is Bad. 

Step 3: Similar to Step 2, activity levels of infrared face image 
are classified and a binary decision matrix, dz, is generated. 

Step 4: Weight matrices my and œz are computed using 
binary decision matrices (dy and dz) and the following three 
conditions: 


(1) Ifdy@) =d;@) = 1, then wy (i) = œwz(i) =0.5. 
(2) If dy (i) = 1 and dz (i) =0, then œy (i) > œwzr(i) and 


lay (i) + 2410) — almeaian| 
ay (i) + ar(i) 





oy) = , (15) 


|a Imedian — 41 (1)| 


16 
ay (i) + ar(i) a 


oi) = 
(3) If dy (i) =O and d;(i) = 1, then my (i) < w;(i) and 


(AV ieia = ay (i)| 





wy (i) = : > (17) 
ay (i) + ar(i) 
lar) + 2ay (i) — aV peaianl 
w(i) = 7 u (18) 
ay (i) + a(i) 
where i is the window count, and aV pegian ANd G1, ian are the 


median values of ay and a; matrices, respectively. Further, 
in all three cases, wy (i) + w;(i) = 1. 


In condition 1, the activity levels of both visible and infrared 
image windows are classified as Good and hence equal weights 
are assigned. Condition 2 states that if the activity level of 
window corresponding to visible face image is classified as 
Good and the activity level of the window corresponding to 
infrared face image is classified as Bad, then higher weight 
is assigned to the visible face image window. In condition 3, 
higher weight is assigned to the infrared face image window 
because 2v-GSVM classifies the activity level of visible face 
image window as Bad and the activity level of infrared face 
image window as Good. 

Step 5: Visible and infrared face images are then fused using 


Fig, (i) = oy (1) Vig,.g (i) + wr (i)Ljg.g (i), (19) 


where F; is the fused subband, j represents the approximate, 
vertical, horizontal, and diagonal subbands, subscript 8 x 8 
denotes that the fusion is performed at window level of size 
8 x 8, and i represents the window count. 

Step 6: Finally, inverse DWT is applied on fused subbands 
to generate the fused multispectral face image, F. Fig. 3 shows 
an example of visible, infrared, and fused face images of an 
individual. 





LWIR 
Face Image 


Fused 
Face Image 


Registered Visible 
Face Image 


Fig. 3. Sample result of the proposed image fusion algorithm. 


4. Match score fusion using DSm theory 


In multimodal biometrics, researchers have proposed sev- 
eral match score fusion algorithms such as AND/OR rule [36], 
Sum rule [36], and SVM fusion [37]. Another mathematical 
paradigm of information fusion is based on the theory of 
evidence. Dempster Shafer theory (DST) [38] based fusion 
algorithm is one example of this paradigm in which uncertain 
and fuzzy information are efficiently fused. In multimodal 
biometrics, it has been shown that DST based fusion algo- 
rithms perform better compared to existing fusion algorithms 
[39]. However, DST has some limitations as reported by Zadeh 
[40-42], Dubois and Prade [43], and Voorbraak [44]. Re- 
searchers have shown that results for DST are not trustworthy 
when conflict between different sources is large. Other limi- 
tations related to Dempster rule of combination are reported 
in Ref. [15]. Recently, DSm theory based fusion algorithm 
[15,16] has been proposed to circumvent the limitations of 
other evidence theory based fusion algorithms. DSm theory is 
a powerful mathematical model for information fusion which 
includes both Bayes theory and DST as special cases. In this 
section, we first present a brief overview of DSm theory and 
hybrid DSm rule of combination [16] followed by the proposed 
match score fusion algorithm. 


4.1. Overview of DSm theory 


DSm theory is a powerful tool for representing and fusing un- 
certain or conflicting knowledge. It can solve complex static or 
dynamic fusion problems using plausible and paradoxical rea- 
soning [15,16]. Since identity verification is a two class prob- 
lem with the classes being genuine and impostor, we explain 
DSm theory for a two class problem. 

Let O = {01, 02} be the frame of discernment which consists 
of a finite set of exhaustive and mutually exclusive hypothesis. 
Hyperpower set of the frame of discernment is defined as D? = 
{Ø, 01, 02, 01 U 02, 0; N 02}. A mapping m(-) on @ is defined 
as m(-) = D® — [0, 1], such that 


X m(A)=1, (20) 


AEDO 


m(A) is called the generalized basic belief assignment (gbba) 
of A and m(@) = 0. Further, a generalized belief function Bel 
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is a mapping function Bel : D® — [0, 1] such that 


Bel(A) = > m(X). (21) 
XCA,XED9 

More specifically, 
(2) 

Bel(A)e;?” [Ey,:](wo € A) =x. (22) 


This equation denotes the degree of belief x of the classifier y 
at time t when wo belongs to A and A € D®. Belief is based 
on evidential corpus Fy, held by y at time ¢. To simplify, gen- 
eralized belief function can also be written as Bel(A). Further, 
generalized belief function, Bel, uniquely corresponds to gbba 
m and vice versa. 

For fusing two information sources, X and Y, the DSm rule 
of combination [16] is defined as 


mm @)(A) = W(A)[S1(A) + S2(A) + S3(A)], (23) 


where M (©) is the model over which DSm theory operates and 
W(A) is the characteristic non-emptiness function of A which 
is 1 if Aø and O otherwise. Sı (A), S2(A), and S3(A) are 
defined as 


sSA= Dd 


(X,YeD®, XAY=A) 


m(X)m2(Y), 


S2(A) = > mı (X) mY), 
(X, Ye, [v=A]V[(ve®)A(A=/;)]) 
$3(A) = 5 mı (X) m2(¥), (24) 


(X,YeD®, XUY=A, XAY eD) 


where J, is total ignorance and is the union of all 0; (i = 1, 2), 
i.e., J, = 0; U 02. ® = {Ọ, p} is the set of all elements of 
D® which are empty under the constraints of some specific 
problem, and ¢ is the empty set. v= u(X) Uu(Y), where u(X) 
is the union of all singletons 0; that compose X and Y. Here, 
Sı (A) corresponds to the classical DSm rule on the free DSm 
model [15], S2(A) represents the mass of all relatively and 
absolutely empty sets which is transferred to the total or relative 
ignorance, and S3(A) transfers the sum of relative empty sets 
to the non-empty sets. An excellent description of DSm theory 
is presented in Ref. [16]. 

Probability based approaches have limitations because they 
deal with basic probability assignment m(-) € [0,1] and 
m(01)+m(02)= 1. Further, DST deals with basic belief assign- 
ment m(-) € [0, 1] such that m(01)+m(02)+m(01 U 02)=1. In 
contrast, DSm theory is more generalized and deals with belief 
functions associated with the generalized belief assignment 
such that m(01) + m(02) + m(01 U 62) + m(01 N 02) = 1. 


4.2. Proposed multimodal match score fusion algorithm 


In this section, we propose a novel match score fusion al- 
gorithm using DSm theory. Fig. 4 shows the steps involved in 
the proposed match score fusion algorithm for a single image. 
Two match scores are computed by matching the global fea- 
tures and local features extracted from probe and gallery face 


images. These two match scores are fused using DSm theory. 
We first define: 


e Frame of discernment: © = {Ogenuines 9impostor}- 
; SDR @ 
e Dedekind lattice: D = {Ovenuines impostor» O genuine U 


impostor» Oceñuine N impostor} 


Let sı and s2 be the two match scores computed from two face 
recognition algorithms. Let us assume that the distribution of 
match scores to an element of DÊ is a Gaussian distribution 


E E TERE S” e (25) 
Peis Piya Ciy Gij v27 p 2 Gij , 


where j4;; and oj; are the mean and standard deviation of the 





ith classifier corresponding to the jth element of D®. We use 
this Gaussian distribution to compute the gbba, 


miC)= DONO sonatie U Dimpostor } —> [0, 1]. 


Since in biometrics, a match score can only belong to genuine 
(genuine) impostor (Oimpostor), or conflicting region (OgenuineN 
impostor) we set mi (Ogenuine U impostor) =0.001 and compute 
the remaining gbbas as follows: 


POS: Hijs Fj) Bij 
D®|-1 i 
ay P(Si, Hij» Fij) Bij 





mi(j) = (26) 


where Bij is the prior of classifier i corresponding to the jth 
element of at Deere U Oimpostor }. We have used Pij as the 
verification accuracy computed on the training database and its 
value lies in the range of (0, 1). Gbbas of the two classifiers 
mı(-) and m2(-) are computed and fused using 


mM fused = M1 ® m2, (27) 


where © represents the hybrid DSm rule of combination defined 
in Eq. (23), Section 4.1. Finally, threshold T is used to classify 
the decision as accept or reject: 


accept if m fused ZT, 


reject otherwise. (23) 


Decision = | 


To extend this algorithm to multispectral face images, we sep- 
arately apply the feature extraction algorithms to the visible 
face image and the infrared face image. As shown in Fig. 5, 
we extract the global facial features using 2D log polar Gabor 
transform [13] and the local facial features using local binary 
pattern [14], and then match these features to compute the cor- 
responding match scores. The two match scores for the visible 
face image are fused using the proposed DSm fusion algorithm. 
Similarly, the two match scores for the infrared face image 
are also fused to generate the fused match score. Finally, the 
fused match scores of the visible face image and the infrared 
face image are combined to compute a composite match score 
for multispectral face images. A decision of accept or reject is 
made using the composite match score. 
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Fig. 4. Block diagram of the proposed DSm match score fusion algorithm in which match scores obtained from two classifiers are fused. 
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Fig. 5. Steps involved in the proposed DSm match score fusion algorithm. In this case, match scores obtained by applying the two classifiers on multispectral 


face images are fused. 


5. Integration of image fusion and match score fusion 


The proposed image and match score level fusion algorithms 
are integrated to further improve the verification performance. 
Fig. 6 shows the block diagram of the integrated multilevel fu- 
sion algorithm. Visible and infrared face images are first fused 
using the proposed 2v-GSVM image fusion algorithm described 
in Section 3. 2D log polar Gabor transform [13] and local bi- 
nary pattern [14] feature extraction algorithms are then applied 
on the fused multispectral face image to extract global and lo- 
cal facial features. Match scores computed by matching these 
features are fused using the proposed DSm match score fusion 
algorithm described in Section 4. 


6. Databases and existing algorithms used for validation 


To validate the proposed fusion algorithms, we used the 
Notre Dame [17,18] and Equinox/NIST [19] multispectral 
face databases. In addition, we used 2D log polar Gabor trans- 
form [13] and local binary pattern algorithms [14] for face 


verification. For comparing the performance of the proposed 
algorithms, we used several existing image fusion and match 
score fusion algorithms. In this section, we briefly describe the 
details of the databases and algorithms used in our experiments. 


6.1. Databases used for validation 


e Notre Dame face database: Notre Dame face database 
[17,18] contains LWIR and visible images from 159 classes 
with variations in expression, lighting, and time lapse. We 
have chosen three visible and LWIR face image with neu- 
tral expression for training database, one neutral visible and 
LWIR image for gallery database, and the remaining images 
comprise the probe data set. Table 1 shows the details of 
training, gallery, and probe images used in the experiments. 

e Equinox face database: Equinox face database [19] contains 
LWIR, medium wave infrared, short wave infrared, and vis- 
ible face images pertaining to 95 individuals. LWIR images 
are captured at 8—12 um, medium wave infrared images at 
3-5 um, and short wave infrared images at 0.9-1.7 um. The 
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Table 1 


Number of visible and infrared image pairs in the training, gallery, and probe 
databases 





Face database Number of visible and infrared image pairs in 





Training database Gallery database Probe database 





Notre Dame 477 159 1815 
Equinox 285 95 18 715 





images are captured under different illumination conditions 
and contain variations in expression and glasses. In our re- 
search, we have used only LWIR images and visible images. 
The number of LWIR and visible images per class vary from 
43 to 516. We chose three visible and LWIR images with neu- 
tral expression and without glasses for training, one LWIR 
and visible image with neutral expression, uniform illumi- 
nation, and no glasses for gallery, and the remaining images 
as probe. 


6.2. Face recognition algorithms 


We first detect the face region from input images. Visible 
face images are detected using the triangle based face detec- 
tion algorithm [45] whereas a thresholding based face detection 
algorithm [4] is applied to detect LWIR face images. Global 
and local facial features are extracted from these detected face 
images using the face recognition algorithms described below. 


e 2D log polar Gabor transform: In the 2D log polar Gabor 
transform based face recognition algorithm, the face image 
is transformed into polar coordinates and textural features 
are extracted using the 2D log polar Gabor transform [13]. 
These features are matched using the Hamming distance to 
generate match scores. 

e Local binary pattern: In this algorithm, a face image is di- 
vided into several regions and weighted LBP features are 


extracted to generate a feature vector [14]. Matching of two 
LBP feature vectors is performed using weighted Chi square 
distance measure algorithm. 


6.3. Existing fusion algorithms 


To compare the performance of the proposed 2v-GSVM 
based image fusion algorithm, we chose image fusion algo- 
rithms proposed by Kong et al. [4] and Singh et al. [10]. Both 
the algorithms use DWT to fuse infrared and visible face im- 
ages. To compare the performance of the proposed DSm based 
match score fusion algorithm, we chose four existing fusion al- 
gorithms, Product rule [36], Sum rule [36], SVM fusion [37], 
and DST fusion [39]. Product rule and Sum rule are based on 
statistical rules, SVM fusion algorithm is a learning based algo- 
rithm, and DST fusion algorithm is based on evidence theory. 


7. Experimental validation of the proposed algorithms 


In this section, we perform the experiments to validate the 
proposed image fusion and match score fusion algorithms. 
Using the training images, we train both 2v-GSVM image fu- 
sion algorithm and DSm match score fusion algorithm. For 
2v-GSVM learning and classification, we used the radial basis 
function (RBF) kernel with RBF parameter as 4. The perfor- 
mance is evaluated in terms of verification accuracy at 0.01% 
false accept rate (FAR). The experimental validation is divided 
into four parts: 


(1) Validation of the proposed 2v-GSVM image fusion 
algorithm. 

(2) Validation of the proposed DSm match score fusion 
algorithm. 

(3) Validation of the integrated multilevel image fusion and 
match score fusion algorithm. 
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Table 2 
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Verification performance of the proposed 2v-GSVM and existing image fusion algorithms at 0.01% FAR 





Face database Recognition algorithm Verification accuracy (%) 





Visible image LWIR image 


Kong image fusion [4] Singh image fusion [10] Proposed image fusion 





Notre Dame 2D log polar Gabor 89.36 88.09 
Local binary pattern 88.20 87.44 
Equinox 2D log polar Gabor 78.91 82.75 
Local binary pattern 76.80 81.52 





(4) Statistical evaluation of the proposed fusion algorithms 
using half total error rate (HTER). 


7.1. Validation of the proposed 2v-GSVM image fusion 
algorithm 


The performance of the proposed image fusion algorithm is 
evaluated using the Notre Dame and Equinox face databases. 
We compared the performance with two existing multispec- 
tral face image fusion algorithms referred to as Kong image 
fusion [4] and Singh image fusion [10]. For evaluation, we 
separately computed the verification accuracies of visible face 
image and LWIR face image using both 2D log polar Ga- 
bor and local binary pattern face verification algorithms. The 
third and fourth columns of Table 2 summarize the verification 
performance of visible face image and LWIR face image, re- 
spectively, using both the verification algorithms. These results 
establish the baseline for evaluating and comparing the per- 
formance of fusion algorithms. We then compute the verifica- 
tion accuracies with the proposed 2v-GSVM multispectral face 
image fusion algorithm and existing image fusion algorithms. 
The results summarized in Table 2 show that the proposed im- 
age fusion algorithm outperforms both the existing fusion algo- 
rithms by at least 3.9% for the Notre Dame database and 4.9% 
for the Equinox database. ROC plots in Figs. 7 and 8 show 
the results for the Notre Dame and Equinox face databases, 
respectively. 

The proposed 2v-GSVM image fusion algorithm performs 
correct classification of multispectral face information at dif- 
ferent levels of granularity which is subsequently used for 
computing the dynamic weights of visible and LWIR face im- 
ages. This granular learning results in better generalization and 
fusion of high entropy visible and LWIR face features. Fur- 
ther, as shown in Fig. 9, fused face images generated from the 
proposed image fusion algorithm provide more invariance to 
illumination compared to the visible images. The fused images 
also provide more distinguishing information compared to the 
LWIR face images. These properties of the proposed 2v-GS VM 
image fusion algorithm lead to improved face verification 
performance. 


7.2. Validation of the proposed DSm match score fusion 
algorithm 


To validate the performance of the proposed DSm match 
score fusion algorithm, we compute the composite match score 
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Fig. 7. ROC plots of the proposed 2v-GSVM and existing image fusion 
algorithms on the Notre Dame face database. Results are computed using (a) 
2D log polar Gabor (b) local binary pattern based verification algorithms. 


as described in Section 4. First the match score is obtained 
from the visible and the LWIR face images. Note that these 
images are not fused. However, as shown in Fig. 5, the fusion 
occurs at match score level. Next, the fused match scores ob- 
tained from each image are combined using the proposed DSm 
algorithm to generate the composite match score. The verifi- 
cation accuracies are summarized in Table 3 and Fig. 10. On 
both the databases, the proposed DSm match score fusion algo- 
rithm yields more than 98% accuracy. On the Notre Dame face 
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Fig. 8. ROC plots of the proposed 2v-GSVM and existing image fusion 
algorithms on the Equinox face database. Results are computed using (a) 2D 
log polar Gabor (b) local binary pattern based verification algorithms. 


database, the proposed algorithm provides at least 1.24% bet- 
ter performance compared to the second best DST based match 
score fusion algorithm [39]. On the Equinox database, the pro- 
posed algorithm provides 1.57% and 3.03% better verification 
performance compared to DST and SVM fusion algorithms, 
respectively. 

The performance of existing match score fusion algorithms 
decreases when two face recognition classifiers yield conflicting 
decisions. For example the classifier using global features may 
make a decision to accept and the classifier using local features 
may make a decision to reject. In such cases, the proposed DSm 
match score fusion algorithm operates on the intersection region 
(genuine N Yimpostor) to make an optimal decision using the 
prior information of classifiers. Existing statistical and learning 
based fusion algorithms including DST fusion algorithm do 
not account for the conflicting region. Hence, the plausible and 
paradoxical reasoning technique of DSm theory provides better 
verification performance. 





Visible 
Image 


LWIR 
Image 


Fused 
Image 


Fig. 9. Results of the proposed 2v-GSVM image fusion algorithm on the 
Equinox face database [19]. 


7.3. Validation of integrated multilevel image and match 
score fusion algorithm 


In previous experiments, we have established that image 
fusion using 2v-GSVM improves the verification performance. 
Also at match score level, the fusion using DSm theory im- 
proves the verification accuracy even when the multispectral 
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Table 3 


Verification performance of the proposed DSm and existing match score fusion algorithms at 0.01% FAR 





Face database Verification accuracy (%) 





Product rule fusion [36] Sum rule fusion [36] 


SVM fusion [37] DST fusion [39] Proposed DSm fusion 





Notre Dame 95.07 97:12 
Equinox 93.20 94.33 


97.46 97.58 98.82 
95.05 96.51 98.08 
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Fig. 10. ROC plots of the proposed DSm match score fusion and existing 
match score fusion algorithms on the (a) Notre Dame face database [18] (b) 
Equinox face database [19]. 


images are not fused. In this section, we integrate both im- 
age fusion and match score fusion algorithms as described in 
Section 5 to evaluate the face verification performance. The 
validation results of the integrated multilevel fusion algorithm 
are summarized in Table 4. The integrated fusion algorithm 
yields 99.91% verification accuracy on the Notre Dame face 
database and 99.54% on the Equinox face database. The 
results also show that the integration of fusion algorithms 


further improves the verification performance by at least 1.08% 
compared to only image fusion or only match score fusion. 
High verification accuracies (> 99.5%) on the Notre Dame and 
Equinox face databases show the robustness of the proposed 
integrated fusion algorithm to the variations in illumination, 
expression, and occlusion due to glasses. The computational 
time of the proposed integrated image and match score fusion 
algorithms including image registration, feature extraction and 
matching is 4.3 s on a P-IV, 3.2 GHz computer under MATLAB 
environment. 


7.4. Statistical evaluation of proposed image fusion and 
match score fusion algorithms 


The performance of a biometric system greatly depends on 
the database size and the images present in the database [46]. 
It cannot be represented completely by ROC plots and verifi- 
cation accuracy. To systematically evaluate the performance, 
Bengio and Mariéthoz [47] have proposed statistical test us- 
ing HTER and confidence intervals [47]. In this section, we 
perform statistical evaluation of the proposed image fusion 
and match score fusion algorithms using HTER. HTER is 
defined as, 


FAR+ FRR 


HTER = (29) 





Confidence intervals are computed around HTER as HTER+<o- 
Za. @ and Zy/2 are computed using Eqs. (30) 
and (31) [47] 





FRR(1 — FRR) 
4:-NG ” 





/ FAR(1 — FAR) 
o= (30) 


4-NI 
1.645 for 90% CI, 


1.960 for 95% CI, (31) 
2.576 for 99% CI, 


Zan = 


NG is the total number of genuine scores and MI is the total 
number of impostor scores. 

Table 5 summarizes the results of this statistical evalua- 
tion using false accept and false reject rates. We have com- 
puted these statistical values at 0.01% FAR. This statistical 
test shows that on a database similar to the Notre Dame with 
any number of classes, HTER of the proposed integrated fu- 
sion algorithm will lie between 0.05 + 0.06 with 95% confi- 
dence. Similar results have been obtained with the Equinox face 
database. 
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Table 4 


Verification performance of the proposed integration of image fusion and match score fusion algorithms at 0.01% FAR 





Face database Verification accuracy (%) 





Proposed 2v-GSVM image fusion 





2D log polar Gabor Local binary pattern 


Proposed DSm match score fusion 


Integrated image and match score fusion 








Notre Dame 95.85 94.80 98.82 
Equinox 94.98 94.71 98.08 
Table 5 


Confidence interval around HTER of the proposed 2v-GSVM image fusion, DSm match score fusion, and integrated multilevel fusion algorithms 














Face database Fusion algorithms HTER (%) Confidence interval (%) around HTER for 
90% 95% 99% 
Notre Dame Image fusion with 2D log polar Gabor 2.08 0.68 0.81 1.07 
Image fusion with local binary pattern 2.61 0.76 0.91 1.19 
Match score fusion 0.59 0.37 0.44 0.58 
Integrated image fusion and match score fusion 0.05 0.10 0.12 0.16 
Equinox Image fusion with 2D log polar Gabor 2.52 0.26 0.31 0.41 
Image fusion with local binary pattern 2.65 0.27 0.32 0.42 
Match score fusion 0.97 0.17 0.20 0.26 
Integrated image fusion and match score fusion 0.24 0.08 0.10 0.13 
8. Conclusion Acknowledgment 


Visible and long wave infrared images provide comple- 
mentary properties which can be combined to improve the 
performance of face recognition. In this paper, we proposed 
image fusion and match score fusion algorithms to fuse in- 
formation obtained from multispectral face images. We first 
apply mutual information based registration algorithm to reg- 
ister multispectral face images and then fuse the images using 
the proposed 2v-granular support vector machine. The fused 
image contains the properties of both visible and long wave 
infrared images and can efficiently be used for face recogni- 
tion. We next proposed the DSm match score fusion algorithm 
to fuse match scores generated from multiple classifiers. These 
match scores can be generated by applying multiple classifiers 
on one image or by applying classifiers on multispectral face 
images. The proposed match score fusion algorithm is based 
on the theory of evidence and performs efficiently even when 
visible and long wave infrared images provide conflicting 
decisions. The proposed image fusion and match score fusion 
algorithms are then integrated to further improve the face 
recognition performance. We validated the performance of the 
proposed image fusion algorithm, match score fusion algo- 
rithm, and the integrated image fusion and match score fusion 
algorithms using the Notre Dame and Equinox face databases. 
Experimental results show that the proposed image fusion and 
match score fusion algorithms outperform existing fusion algo- 
rithms. Results and statistical evaluation further show that the 
proposed integrated image and match score fusion algorithm 
yield best performance among all the proposed and existing 
fusion algorithms. 


The authors would like to thank CVRL University of 
Notre Dame and Equinox Corporation for providing the face 
databases used in this research. 
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